Skip to content

feat(otel): Add per-PR OTel conformance test suite - #630

Merged
ayushiahjolia merged 2 commits into
mainfrom
otel-conformance-tests
Aug 18, 2026
Merged

feat(otel): Add per-PR OTel conformance test suite#630
ayushiahjolia merged 2 commits into
mainfrom
otel-conformance-tests

Conversation

@ayushiahjolia

@ayushiahjolia ayushiahjolia commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

aws/aws-durable-execution-conformance-tests#90

Description

Adds a per-PR OTel conformance suite for the Java SDK, matching the per-PR coverage the non-OTel conformance-tests/ module already has.

  • New conformance-tests-otel/ module: handlers ported from aws/aws-durable-execution-conformance-tests's Java OTel example project (otel-invocation, otel-execution, and otel-long-running in short mode), plus template.yaml and template-long-running.yaml.
  • New .github/workflows/otel-conformance-tests.yml: runs on PRs touching sdk/**, otel-plugin/**, or conformance-tests-otel/**, across the X-Ray, Dash0, Datadog, and S3-collector backends.
  • Self-contained by design: deploys and validates directly rather than calling the shared orchestrator, so it needs no secrets beyond whatconformance-tests.yml/e2e-tests.yml already use, plus new Dash0/Datadog credentials for those two backends.

Demo/Screenshots

N/A

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change

Testing

Unit Tests

Have unit tests been written for these changes? N/A

Integration Tests

Have integration tests been written for these changes? N/A

Examples

Has a new example been added for the change? (if applicable) N/A

Comment thread conformance-tests-otel/template.yaml Outdated
Comment thread .github/workflows/otel-conformance-tests.yml Outdated
Comment thread .github/workflows/otel-conformance-tests.yml Outdated
@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the otel-conformance-tests branch from 51bf2d8 to 609d7ce Compare August 17, 2026 21:32
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 17, 2026 21:32 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 17, 2026 21:32 — with GitHub Actions Inactive
Comment thread .github/workflows/otel-conformance-tests.yml Outdated
Comment thread .github/workflows/otel-conformance-tests.yml Outdated
Comment thread pom.xml Outdated
@github-actions

This comment has been minimized.

Comment thread conformance-tests-otel/pom.xml Outdated
@github-actions

This comment has been minimized.

Comment thread .github/workflows/otel-conformance-tests.yml Outdated
@github-actions

This comment has been minimized.

Comment thread pom.xml Outdated
Comment thread .github/workflows/otel-conformance-tests.yml
@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the otel-conformance-tests branch from 727842b to 92b7102 Compare August 18, 2026 19:20
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 18, 2026 19:36 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 18, 2026 19:36 — with GitHub Actions Inactive
Comment thread .github/workflows/otel-conformance-tests.yml Outdated
Comment thread conformance-tests-otel/template.yaml
@github-actions

This comment has been minimized.

Comment thread .github/workflows/otel-conformance-tests.yml Outdated
@github-actions

Copy link
Copy Markdown

Claude AI review

This PR adds a standalone conformance-tests-otel/ module (19 OTel handlers + long-running variants, two SAM templates) and a new otel-conformance-tests.yml workflow. The Java handlers are correct against the current SDK API surface: retry handlers use the 1-based StepContext.getAttempt() with fixedDelay(2, …) correctly, Otel14MapFailure relies correctly on the lenient allCompleted default and throws on !allSucceeded(), Otel8HandledFailure catches the rethrown step exception before the recovery step, and generic result types use TypeToken/Class overloads that exist. The standalone (non-reactor) pom.xml is intentional and documented, and the lack of unit tests is consistent with the existing conformance-tests/ module and AGENTS.md/CONTRIBUTING.md (conformance handlers are deploy-only).

One confirmed, CI-breaking issue: the reusable-workflow reference still contains the literal placeholder @<ORCHESTRATOR_SHA> (see .github/workflows/otel-conformance-tests.yml:64). Because the workflow triggers on pull_request/push to sdk/**, otel-plugin/**, conformance-tests-otel/**, and pom.xml, merging as-is causes every matching PR to fail this check with an "invalid reusable workflow reference" error. It is acknowledged by the adjacent TODO, so the concrete gate is to pin a real SHA before this leaves draft/merges.

Minor (not blocking): OtelConformanceHandler's class Javadoc states the otel-execution suite runs "against the X-Ray backend only," but template.yaml deploys the execution functions in the same per-backend stacks as the invocation suite (Dash0/Datadog/S3 collector included), so that comment appears stale/inaccurate. Worth correcting to avoid misleading maintainers, but it has no runtime effect.

Residual test risk: none of the new handlers are exercised by local/replay tests (by design); their behavior is only validated by the external orchestrator once the SHA is pinned, so regressions in the ported scenarios would surface only in the deployed conformance run.

Reviewed commit 92b7102410345a19044b74782c7952056d32541b. Workflow run

Comment thread .github/workflows/otel-conformance-tests.yml Outdated
@github-actions

This comment has been minimized.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
# Build the handlers from this repo's checked-out module instead of the conformance repo's
# bundled examples/java. Path is relative to the conformance workspace where the SDK is
# checked out (.build/durable-sdk).
examples_dir: .build/durable-sdk/conformance-tests-otel

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia marked this pull request as ready for review August 18, 2026 22:43
@ayushiahjolia
ayushiahjolia deployed to ai-pr-review-runtime August 18, 2026 22:48 — with GitHub Actions Active
@ayushiahjolia
ayushiahjolia merged commit ae9e395 into main Aug 18, 2026
16 of 28 checks passed
@ayushiahjolia
ayushiahjolia deleted the otel-conformance-tests branch August 18, 2026 22:54
@github-actions

Copy link
Copy Markdown

Codex AI review

No actionable findings. Residual risk remains in the live SAM deployment and external OTel backend matrix, which were not executed under the read-only review constraints.

Reviewed commit f28b9a3401310d51b5b7f67e4d339d02abb5100e. Workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants